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ABSTRACT 

Under the name of Citizen Science, many innovative practices in which volunteers partner with 
scientist to pose and answer real-world questions are quickly growing worldwide. Citizen Science 
can furnish ready made solutions with the active role of citizens. Flowever, this framework is still 
far from being well stablished to become a standard tool for Computational Social Sciences 
research. We present our experience in bridging Computational Social Sciences with Citizen 
Science philosophy, which in our case has taken the form of what we call Pop-Up Experiments: 
Non-permanent, highly participatory collective experiments which blend features developed 
by Big Data methodologies and Behavioural Experiments protocols with ideals of Citizen 
Science. The main issues to take into account whenever planning experiments of this type are 
classified and discused grouped in three categories: public engagement, light infrastructure and 
knowledge return to citizens. We explain the solutions implemented providing practical examples 
grounded in our own experience in urban contexts (Barcelona, Spain). We hope that this work 
serves as guideline to groups willing to adopt and expand such In-vivo practices and opens the 
debate about the possibilities (but also the limitations) that the Citizen Science framework can 
offer to study social phenomena. 
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1 INTRODUCTION 


The relationship between science and society has always been an importan t aspect t o consider when on e 
tries to understand how science advances and how research is performed (|Latour[ fT998| [Kuhn l |2012|) . 
The general public has however been m ostly left out of its methodology and creation processes (ILatour 


2006[ [Callon and Rabeharisoa 

only li 


20031. Citizens are generally considered as passive subjects to whom 
nished results can be presented in the form of simplified statements, 


yet paradoxically, we 

implicitly ask them to support and encourage research. The acknowledgement of this Ivory Tower problem 
has recently opened new and exciting opportunities to open minded scientists. The advent of digital 
communication technologies, mobile devices and the Web 2.0 is fostering a new kind of relation between 
professional scientists and dedicated volunteers or participants. 
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Under the name of Citizen Seienee (CS), many innovative praetiees in whieh “volunteers partner with 
seientist to answer and pose real-world questions” (as stated in the Cornell Orni thology Lab webpage, 


one of the preeursors of CS praetiee s in 19 80s) are quiekly growing worldwide (Hand 2010; Bonney 


et al. . 2014, Gura 2013; |Editorial 20151. Formally, CS has been reeently defined by the Soerentrze 


ite Paper by a ‘general publre engagement in seientifie researeh aetivities when eitizens aetively 
eontribute to se ienee either with their intelleetual effort or surrounding knowledge or with their tools 
and resourees” (|Socientize[ |20141). This open, networked and transdiseiplinary see nario, favour s a more 
demoeratie rese^eh thanks to amateur or non professional seientists’ eontribution (|Irwin[| 19951). During 
last few years, relevant results h ave been published in high impaet j ournals sueh as Nature or Seienee 

~ The hidden power of thousands 

with eomparable (or even better) 


20T4t[Gural |2m3). 
•ent rn many fields. 


by using partieipatory praetiees (Bonney et al. 
of hands working together is making itself app^ent 
performanee to expensive supereomputers when used to analyse/elassify astronomieal images (|Lintott 
|et al4 [2008]), to reeonstruet 3D brain maps based on 2D images (jKiin et al4 |2014|), or to find out 
stable bromoleeular struetures (|Cooper et al.l |20101) just to name very few large rmpaet eases. Citizen 
eontributions ean also have a drreet impaet ni so erety by for instanee helping to ereate exhaustive and 
shared geo-loealised datasets ( |Lauro et al.[ |2014| ) at a density level unattainable for the vast majority of 
private sensor networks (and at a mueh redueed eost) or by eolleetively gathering empirieal evidenees 
to foree publie administ ration aetion (for example the shutdown of a noisy faetory loeated in the eity of 
London |Haklayl 12013])). Most aetive volunteers ean therefore eontribute by providing ex perimental data 
and widen the researehers reaeh, raise new questions and eo-ereate a new seientifie eulture (|Latour[|2006 
McQuillani[20T4l). 


Computational Soeial Seienees (CSS) is a multidiseiplinary field at the interseetion of the soeial, 
eompu tational and eomplexity seienees whieh s ubjeet of study is the human interaetions and soeiety 


itself (Lazer et al. , 2009, Cioffi-Revilla [2010). However, CS praetiees in this eontext remain vastly 


unexplored when eompar ed to others fie l ds sueh as Environmental Seien e es whieh already c arry a long 


story on their shoulders (Russell 2014 


Dickinson and Bonneyl |2015[ |Silvertown| |20b9| ). Attempts 


where participation of or dinary citrzens has played an important role can be found within fields like 


Experime ntal Economics (| Hoggatti|1959|), financial trading floors design ([Cueva et al. 1120151), or human 
mobility ([Yoshimura et al.[ |2014|). Also works on the emergence of cooperation ([Grujic et al.] [20101) 
and the dynamrcs of socral mteractrons ([Weiss et al.[120141) are noteworthy. All these experiments yielded 
relevant scie ntific o u tcome s wit h well-stablished and rob ust protocols within Behavioural Sciences (see 
for instance [Kagel] ([19971) ^^id ([Margolin et al.[ [2013[)) but unfortunately they stay on the very first 
level of CS scale (jliaklay] [2013[), where citizens are involved only as sensors or volunteer subjects of 
certain experiment in strictly controlled environments, be ing their participation and potential only partially 
unleashed. A possible way out was already provided by [Latour[(2006[ 2009]) when proposing collective 
experiments in which the public becomes a driving force of the research. Researchers in the wild are then 
directly concerne d with the knowledge they produc e because they are both the objects and the subjects 
of their research (ICallon and Rabeharisoal 2003]). Some interesting research initiatives have emerged 


along these lines to run mas sive experiments m collaboration with a CS foundation such as Ibercivis 
([Gracia-Lazaro et al.[ [2012]) or through online platforms such as Volunteer Science online platform from 
Lazer Lab. More radical initiatives also consider the collaboration with artists and have been carried out 


in museums or exhibitions and as big performances (Perello et al.[ 2012; Cattuto et al.[ 2010). 


CSS research has also been recently working within the so called Big Data paradigm (Schroeder 2014 
Conte et al.[ [2012[). A lot has been said about it and the possibilities it offers to society, industry ani 
researchers. Smart Cities density urban areas with all kind of sensors and integrate the information with 
a wide collection of datasets. Mobile devices represent also a powerful tool to monitor on real time user 
related stats such as health and great businesses opportunities are already being foreseen by companies. 
However, these approaches again treat citizens as passive subjects from whom one records private data in 
an unconsented way, with the aggravated problem that the unaware producers of this data (i.e., citizens) 
lose control of its use, exploitation and analysis. Concerning this last aspect, the validity of the conclusions 
drawn from the analysis of such datasets are still today a subject of discussion, mainly due to the poor 
control on the data gathering process (by the public in general and by scientists in particular), inherent 
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population and sampling biases (Lazer et al. 20141 and lack of reproducibility among other systemie 
problems (|Boyd and Crawforcl[ |2U12|). Last but not least, the Big Data paradigm has so far failed in 
providing soeiety with the neeessary publie debate and transparent praetiees from the bottom-up approaeh 
it preeonizes: It eurrently relies on huge infrastruetures only available to private eorporations, whieh may 
not have eoineident objeetives with those of researehers and eitizenry; and provide eonditioned aeeess to 
the data eontents whieh, in addition generally eannot be freely (re)used without filter. 

Our purpose here is not to diseuss some of the Big Data inherent problems. The approaeh we present 
aims however to explore the potential in blending interesting features reeently developed by Big Data 
methodologies with the ambitious and demoeratie ideals of CS. Publie partieipation and seientifie 


empowerment induce a level of (conscious) proximity with the subjects of the experiments that can be 

a very valuable source of high quality data (|Vianna et al. 

|2014 Rossiter et al. 

2015 

), or at least, of 

non-conflictive information with regards to data anonymity ( 

Bowser et al. 

2014 ), that may be considered 


and ethieal issu es arisen when eolleeting data from digital soeial platforms while keeping high standards 
of partieipation (|Schroederl|2014[|Resnik et al.[|2015]|Riesch and Potter[|20131). Moreover, CS projeets 
V of soeial pL 


use a vast variety of soeial platforms for optimizing dissemination, to eneourage and ine rease partieipation 
and sometimes to develop gamification strategies (]Prestopnik and Crowston[ |2011|) to reinforee sueh 
engagement. The so-ealled Seienee of Citizen Seienee studies the e mergent partieipatory dynamies in this 
elass of projeets (|Sauermann and Franzoni[|2015t|Curtis[ 12015]) so that this also opens the door of new 
eontexts to study soeial phenomena. 

The open philosophy at the heart of CS methods, like open-data lieensing and e oding, ean also e learly 


improve seienee-soeiety-poliey interaetions in a demoeratie and transparent way (Petersen 1984 1 . The 


CS approaeh simultaneously represents a powerful example of Respo nsible Researeh and In novation 


(RRI) praetiees ineluded in the EU Horizon 2020 researeh programme (EUCommision 2014) and the 


Quadruple Helix Model where government, industry, aeademia and eivil partieipants work together to eo 
ereate the futur e and drive struetural ehanges far beyond the seope of what any one organization or person 
eould do alone ( Chesbrough[ 2006). Along this frame, we eonsider that the CSS potential when assuming 


CS methods is vast sinee its subject of study are the citizens themselves. Therefore, their engagement 
with projects studying their own behaviour is highly likely since it has an immediate impact in their 
daily life. As a result, large motivated communities and scientists can work hand in hand to tackle CSS’ 
arising challenges but also collectively circumvent potential side effects. The possibility to reach wider 


biases dHenrich et al. 

12010) and problems of reproducibility present in Social Sciences empirical studies 

(Collaboration 

2015 

Bohannon, 2015). Another important advantage of working jointly with different 


subjects from their natural urban environment -where socialization takes place - but do it in a transparent. 


fully c onsented and enriche d way thanks to the active participation of citizens ( Gallon and Rabeharisoa 


20031 [Perello et al.l 120121). Such practices and methodologies, however, are still far from being wel 

— k — 


stablished to become a standard tool for CSS research. 


The main goal of this paper is precisely to motivate the somewhat unexplored adoption of CS practices 
by CSS research activities. Along this arduous task, we limit ourselves to a reformatting of existing 
standard experimental strategies and methods in science through what we call Pop-Up Experiment 
(PUE). Such a concept has been shaped from our own lessons gained while running experiments in 
public spaces of the city of Barcelona (Spain). Section introduces this very flexible solution to make 
possible a collective experimentation and discusses its toee essential ingredients: public engagement, 
adaptable infrastructure, and knowledge return to citizens. Einally, Sect, [^concludes the manuscript with 
a discussion of what we have presented so far jointly with some consi^rations about the future of CS 
practices within the CSS research. 
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2 A FLEXIBLE SOLUTION FOR CITIZEN SCIENCE PRACTICES INSIDE CSS: THE 
POP-UP EXPERIMENTS 

2.1 CONTEXT AND MOTIVATION 


During the last four years, the Bareelona Couneil and its Direetion of Creativity and Innovation in 
eollaboration wit h several organisatio ns have aimed to explore the possibilities to transform the eity into a 
publie living lab (|Chesbrough[ |2006l) where new ereative teehnologies ean be tested and new knowledge 
ean be eolleetively eonstrueted. This has been done through the Bareelona Lab platform and one of its 
most preeminent aetions has been to establish CS praetiees in and with the eity. The first task led by some 
of us has been to ereate the Bareelona Citizen Seienee Offiee and to build a eommunity of praetitioners 
where most of the CS projeets from several researeh institutions in Bareelona eould eonverge. The Offiee 
serves as meeting point for CS projeets, where researehers ean join forees, experienees and knowledge 
and also where eitizens ean eonneet with these initiatives in an easy and effeetive way. The seeond task is 
direetly linked with the objeet of this paper and was eoneeived to test how far the publie administr ation 


ean go in opening up their resourees to eolleetively run seientifie experiments (Perello et al.| 2012). The 


CS toolbox elearly provides a perfeet framework for the design of publie experiments and the exploration 
of the emergent tensions and the problematie issues faeed when running publie living labs. Furthermore, 
the involvement of the City Couneil provided us with the opportunity to embed these experiments into 
important and massive eultural events, whieh eonstitute a perfeet environment to reinforee the openness 
and the transpareney in our researeh proeess in front of the soeiety or at least in front of the Bareelona 
eitizens. 


We have been eondueting several experiments to bring CS ideals into praetiee and test its potential in 
urban eontexts. In eontrast with existing environmental CS projeets in other eities sueh as London or 
New York (with eivie initiatives sueh as Mapping for Change or Public Lab, respeetively) we foeussed 
our attention to CSS related problems with the aim of providing unexplored relations between eity, 
eitizens and seientists. More speeifieally, this rather wild testing eomprised seven different experiments 
performed between 2012 and 2015 in three different topies addressing diverse questions: human mobility 
(How do we move?), soeial dilemmas (How eooperative are we?) and deeision-making proeess (How 
do we take deeisions in a very uneertain environment like finaneial markets?). These are summarized 
in Tab. and fully deseribed in Seetion ^ (Material and Methods). Other eommon points of these 
experiments are the important number of volunteers partieipating (up to 541 for a single experiment, 1,255 
in total) and the eonsequent large number of reeords (up to 18,525 deeisions for a single experiment, 
55,390 reeords in total), despite a rather limited budget assoeiated to the experiments (from 1,000 to 
4,000 EUR/experiment, and around 2,200 EUR/experiment on average). Despite dealing with different 
researeh q uestions, eommon pr oblems (related to CS praetiees but also CSS researeh studying human 


behaviour (Conte et al.[ 20121) have been identified and solutions have been developed in all eases 


to overeome them. Some of tne implemented solutions have been sueeessful some others not, but all 
experienees have shaped the eoneept and the proeess for an experimentation in CSS researeh eonsistent 
with CS ideals. 


2.2 POP-UP EXPERIMENT DEFINITION AND UNDERLYING PROCESS 

The generie definition of a pop-up aeeording to the Cambridge dietionary is “Pop-up (adj.): used to 
deseribed a shop, restaurant, ete. that operated temporarily and only for a short period when it is likely 
to get a lot of eustomers.” Erom early beginning we thought that this deseription fully fits into our non¬ 
permanent but highly partieipatory experimental set up when applying CS prineiples to Computational 
Soeial Seienees researeh in urban eontexts. The parallelism is very illustrative to better understand a 
mueh more formal definition we use to deseribe our approaeh from a theoretieal perspeetive. It is based 
on the expertise gained from the seven experiments earried out during the last four years and reads: 
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A Pop-Up Experiment (PUE) is a physieal, light, very flexible, highly adaptable, reprodueible, 
transportable, tuneable, eoheetive, partieipatory and publie experimental set-up for urban eontexts 
that (1) applies Citizen Seienee praetiees and ideals to provide ground-breaking knowledge and (2) 
transforms the experiment into a valuable, soeiahy responsible, eonsented and transparent experienee 
to non-expert volunteered partieipants with the possibility to build the urban eommons arisen from 
faets-based effeetive knowledge valid for both eities and eitizens. 


In our ease, we apply this eoneept to CSS aiming to answer very eonerete researeh questions with 
the partieipation of larger populations than those from Behavioural Experiments. The arising researeh 
proeess from PUEs ean be synthesized in the flow diagram from Eig. The whole proeess starts from a 
researeh question and/or a soeietal ehahenge that may be promoted by eitizens, seientists but also private 
organisations, publie institutions or eivie movements. The initial impulse helps to ereate an adequate 
researeh group whieh will need to be multidiseiplinary if it is to taekle a eomplex problem with many 
interwined issues. The group then eo-ereates the experiment both eonsidering the experimental set-up 
and the unavoidable publie engagement tasks. The experiment is then earned out and data is eolleetively 
generated (erowsoureed) under the partieular eonstrains of publie spaees whieh not only depend on the 
eonditions designed by the seientists but also on many other praetieal limitations. The data is afterwards 
analyzed using st andard seien t ifie methods but non - profes sional seientists are also invited to eontribute 


on speeifie tasks (Kim et al., 2014 Lintott et al. 2008) or by using other non-standard strategies in 


the data exploration (|Cooper et aid |2010p. These two eontributions by volunteers eonform what we 
eall Distributed Intelligenee and generate results that ean hardly be matehed by eonventional eomputer 
analysis. In any ease, obtained results ean take many forms aeeording to the audienee being addressed, 
from a seientifie paper to personalized reports readable to any eitizen or even reeommendations valid for 
polieymakers at a eity level. Einally, the whole proeess ean generate the neeessary inertia to promote and 
faee a new soeietal and seientifie ehallenge or an existing need along the same seheme. 


The PUE solution also represents a middle ground between Behavioural Seienee experiments and Big 
Data methodologies. To better understand in whieh eontext the PUEs we propose ean be plaeed, Eig. 
eompares the different approaehes eonsidered using a radar ehart that qualitatively measures with three 
degrees of intensity (low, medium and high) six different aspeets eharaeterizing eaeh experiment type. We 
observe that the three different approaehes eover different areas. Behavioural Experiments and Big Data 
have a limited overlap while the PUEs share several aspeets with the former two. One might argue that 
the excess of openness of the CS eonstitutes a severe limitation to raise objeetivity in eomp arison to the 
solid experimental protoeols in Behavioural Seienees (jWebster and SelH|2()14^|KageH|1997i) but it is also 
true that the highly partieipatory nature of CS ean be very effeetive to reaeh a more realistie speetra of 
pop ulation and a larger samp le thus obtaining more general statements with a stronger statistieal support 
(see |Margolin et al.j (|2013 ) for alternative and eomplementary methods). Sinee it is direetly attaehed 
to real-world situations, it avoids the somehow exelusive and sometimes distorted spaee of the in-vitro 
(or ex-vivo) laboratory experienees. It also brings additional values to the more elassie lab-in-the-field 
experienees in Soeial Seienees whieh generally limit interaetion among subjeets and seientists as mueh 
as possible. On the other extreme, CS praetiees will never be able to eompete in terms of data quantity 
with the Big Data world but this ean be eompensated. A better understanding of the mass of volunteers 
involved and improved knowledge of its peeuliarities helps in avoiding possible biases. Eurthermore, the 
aetive nature of PUEs allow the possibility of tunning some eonditions of the experiments to explore 
alternative seenarios. The PUEs ean indeed be an alternative to the eontroversial virtual labs in soeial 
networks and mobile games whieh have obtained for instanee interesting results on emotional eontagion 


with experiments on the Eaeebook platform (Kramer et al. 2014) not w ithout an inten se publie debate 


on ethieal and privaey issues on the way how the experiments were made (Gleibs, 2014). 


We think that PUEs ean be an essential approaeh for empirieally testing the abundant statements of 
CSS eomplementary to lab-in-the-field, virtual labs and in-vitro experienees. To beeome so, we have 
identified the main obstaeles that hinder the development of CS initiatives with respeet to other forms of 
soeial experimentation. They ean be grouped into three groups: Infrastrueture, Engagement, and Return. 
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In the following, we detail eaeh of the obstacles and illustrate the solutions that PUEs offer together with 
practical examples applied to each case. 


2.3 LIGHT AND FLEXIBLE EXPERIMENTAL INFRASTRUCTURE 


By infrastructure we understand all necessary logistics to make the experiments possible. In a broad 
Citizen Sci ence context, t he ne cess ary elements differ from orthodo x scientific infrastructure. As 
discussed in|Bonney et al.|(|2014|) and|Franzoni and Sauermannl(|20141), they include other tools, other 
technical support and other spaces. Second section of Tabjl lists some of the elements we have deemed 
capital to satisfactory collect reliable data. PUEs should be aesigned favouring scalability in the sense of 
allowing the increase of the population sample size or the repetition of the same experiment in another 
space easily. To make this possible, the experiments must rely on solid and well tested infrastructures, 
with an appealing user experience to avoid frustration by participants. When considering the experimental 
setup, we used several strategies to foster participation and ensure the success of the experiments. 


Eirst, we physically set up the PUEs in very particular contexts in urban areas and, in all cases, we 
placed them in crowded (moderately to highly dense) places to easily reach volunteers. In other words, we 
preferred to go where citizens were instead of encouraging them to come to our labs. To make this possible 
the City Council has for instance offered specific windows in a couple of festivals as hosting events (Bee- 
Path(l), Bee-Path(2), Cooperation(l), Mr. Banks and Dr. Brain experiments). This meant that we had to 
adapt to these out-of-the-lab and in-vivo specific contexts, the logistics and the composition of research 
teams thus unavoidably becoming more diverse, complex, heterodox and highly multidisciplinary. In 
collaboration with the event organizers, we then habilitated a specific space of reduced dimensions for the 
experiment, where the volunteers (their typology was different in each case) could participate through a 
recording device. 


Second, PUEs demands that devices used by participants to collect and manipulate data, either in a 
active or passive way, must be familiar to them. In our experiments we have designed specific software 
running in laptops (Cooperation(l) and Cooperation(2) experiments), mobile phones (Bee-Path(l) and 
Bee-Path(2) experiments) and tablets (Mr. Banks and Dr. Brain experiments). However, it may also be 
possible to use video cameras, photo cameras, or any other sophisticated device as long as participants 
easily get familiarized with it after few instructions or a tutorial. This sort of infrastructure is in the end 
what allows to carry out experiments in a participant’s quotidian (not strange or in-vivo) environment. 
Initially, we overlooked this aspect in the set-up design and allocation of resources, but having friendly 
user interface is important to allow people to behave normally. Similarly, both instructions and interface 
should be understandable and manageable for people of all ages. 


Third, in order to study social behaviour in different environments, the experiments need to be adaptable, 
tunable, transportable, versatile and easy to set up in different places. All devices mentioned before fulfil 
this requirement as well. 


Eourth, PUE are typically one-shot since we are hosted in a festival, a fair or a class-room, which 
means concentrated in time (duration from 0.5 to 2 days) and no chance for a second shot. All collectable 
data could be threatened if something goes wrong. A strong beta testing and defensive programming is 
imperative to ensure that collected data is certainly reliable. It is also necessary to anticipate potential 
problems: One must be flexible enough to be able to retrace alternative research questions on the fly if 
PUEs location and conditions are not fully satisfactory to respond to the initial research purposes. 


Einally, numbers matter and experiments must reach enough statistics to perform rigorous analysis and 
this needs to be carefully taken into account during the design phase of the experiment (see Eig. [T]). 
Typically, the more expensive devices are, the less number of data collectors you can have so the capacity 
to collect data is affected, and this is not very effective strategy in a rather short time one-shot event. 
Therefore, cheap infrastructures favour scalability at the end. Alternatively new collaborators needs to 
be found or an extra effort is required for finding a sponsorship (for instance related to science outreach) 
which in any case will make the experiment preparation phase more complicated. Scalability is interesting 
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indeed but it has its side-effeets as well: Relying on user-provided infrastruetures sueh as smart phones 
ean influenee a lot the quality of the data and its normalization (one of the eentral problems related to 
the Big Data paradigm). In the Bee-Path(l) experiment, where we have used the GPS of partieipants 
own smartphones, the eleaning proeess has been far more laborious than in the other eases due to this 
eireumstanee (see numbers in Tab. [T}. In eontrast, Cooperation(I), Cooperation(2), Mr. Banks and Dr. 
Brain experiments, where we designed and programmed the software of the partieipants experienee, have 
not required mueh posterior treatment. 


2.4 PUBLIC ENGAGEMENT TOOLS AND STRATEGIES 


The PUEs are physieally based and rooted on partieular, temporal and loeal eontexts. This delimited 
framework allows to eoneentrate dissemination resourees and efforts on a given spot over a eertain time, 
whieh inereases the effeetiveness and effieieney of the eampaign in terms of workforee and budget. 
Additionally, the one-shot nature of PUEs allows us to avoid the problem of keeping partieipants engaged 
in an aetivity spanning long periods of time, but eonsequently relies eompletely on eonstantly renewing 
the base of partieipants (whieh may require higher dissemination). To this end, the initial aetion was the 
ereation of a eensus of volunteers shared by all researeh group members of the Bareelona Citizen Seienee 
Offiee. 


Another faetor of eapital importanee relates to the eontaet between researehers, organizers and eitizens 
in the set-up of the experiment. This allows for a niee dialogue and exehange of views that in turn helps in 
framing the seientifie question being studied, as well as developing possible improvements for future 
experienees. This ean be done by stimulating partieipants’ euriosity on the experiment and researeh 
assoeiated to it. The researeh question should be foeussed and understandable for anyone who is not 
expert on the field. 


To engage in sueh a dialogue, however, requires eertain steps. Eirstly, one needs to attraet potential 
partieipants with an appealing set-up. This ineludes loeation in the physieal spaee, but also an effeetive 
diffusion eampaign on the days prior to the experiment (see preparatory aetions in Tab.[^. It is important 
to offer a harmonized design with eommon themes that eitizens ean relate to the experiment. To make 
our material appealing, we eollaborated with an artist (Bee-Path(l) and Bee-Path(2) experiments) and a 
graphie designer (Cooperation(l), Cooperation(2), Mr. Banks(l), Mr.Banks(2) and Dr. Brain experiments) 
whose main eontribution was the ereation of eharaeters assoeiated to eaeh experiment. The funetion of 
these eharaeters, elose to the world of eartoons, was to attraet the publie attention but also to present 
the experiment as an attraetive game sinee one of the most powerful elements to engage people in an 
aetivity is the expeetation to have fun. It is eertainly possible to maintain seientifie rigour while using 
gamification strategi es to ereate a playab le atmosphere for the study, thus transforming it into a more 
eomplete experienee (|Morris et al.l,|2013|). Moreover, aetors were used as human representations of these 
eharaeters (Mr. Banks and Dr. Brain). Ine aetors were indeed one important element to bring on-site 
attraetiveness to the experiment, along with: a large team of seientists and faeilitators present (up to a 
rotating team of 10 people), an optimum and visible loeation inside the event spaee and material/deviees 
to promote the experiment sueh as sereens to visualize on real-time the results of the experiments or 
promotional material (flyers and merehandising). Based on the experienee, we have optimized all these 
ingredients and we have even ineluded some elements of seenography in the last experiment done within 
the Bareelona DAU Board Games Eestival (Dr. Brain). 


But getting enough people is not enough. Additionally, we must also seek for universality in our 
population sample. The experiment must be designed in a way that people from all ages and eonditions 
ean really partieipate. Besides, a PUE has to be transportable with a minimal eost onee it has been 
implemented one time in order to be reprodueed in different environments (whieh may favour eertain types 
of population). As an example ease, Cooperation(l) and Cooperation(2) experiments are very illustrative. 
In Cooperation(I) experiment, we found how different age groups, speeially ehildren ranging from 10 to 
16 years old, behaved in a different way and eooperated with different probability. Apparently ehildren 
were more volatile and less prone to eooperate than the eontrol group in a repeated Prisoner Dilemma. 
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Fifteen months later we repeated the experiment in a Seeondary sehool (12-13 years old, Cooperation(2) 
experiment). On one side, the results in this ease showed the same levels of eooperation than the eontrol 
group in Cooperation(l) experiment. However, on the other side, same volatile behaviour, more than 
Cooperation(l)’s eontrol group, was again found. Therefore, thanks to the repetition of the experiment we 
rejeeted the early idea of different eooperation levels in ehildren, but at the same time it strengthened the 
ehildrens volatile behaviour elaim. 


2.5 OUTCOME AND RETURN TO THE PUBLIC 


Last but not least eome faetors related to the management of the aftermath of the experiment (fourth 
seetion of Tab. [^. The PUEs as we have implemented them are intrinsieally eross-diseiplinary and involve 
a large number of agents and institutions whieh in turn may have diverging interests and expeetations 
regarding the outeome of a partieular experiment. Any sueeessful PUE must be able to aeeommodate all 
these interests and ereate positive eollaboration environments where all aetors eontribute in a eonformable 
way. 

Organisers of a festival will for instanee find in the PUEs an innovative format with partieipatory 
aetivities to add to their program (Cooperation(l), Mr. Banks and Dr. Brain experiments). PUEs also 
ean be a transparent and pro-aetive data gathering system to provide information and analysis about the 
event itself, useful for its planning and improvement. The Bee-Path(l) experiment studied how visitors 
moved around a given spaee and obtained faet-based arguments to improve the spatial distribution in 
future editions of the fair where it was developed. The eity and loeal administration will find in PUEs an 
innovative way to establish a direet eontaet with their eitizens, to eo-ereate new knowledge valid for the 
eity interests and eventually to generate a eensus of highly motivated eitizens ready to eollaborate in these 
kind of aetivities. Seientists will obviously try to publish a new researeh paper with the data gathered. 


All these e xpeetations are very different and should organieally eonverge to suoeessfully run a eolleetive 
experiment (|Latour[|2006j). However, we should also not forget to earefully inelude the expeeted return of 
our eentral aetors: the dedieated volunteers. Their eontribution is essential in CS praetiees (|Irwin[|1995|) 
and it is therefore eompletely fair to argue that eitizens aeeepting to partieipate in these initiatives need to 
elearly see a benefit from their perspeetive eomparable (albeit different) to that of City Couneil, festival 
organisers, seientists or any other eontributor. Moreover, the high eoneentration in spaee and time jointly 
with the intense publie exposur e of the PUE s inerease even more the vol unteer’s expeetations eompared 

ILintott et al.l |2008|). The faee-to-faee relationship 
the 


to other ordinary eases in CS (Kim et al.| 2014 


established between researehers and eitizens in all the PUEs we have been running eertifies that this is a 
very delieate issue that needs to be managed with real eare. 


Any PUE should manage expeetations at three different time seales: Short (during and right after the 
experiment), medium (a week or a month after), and long times (months or even a year later). In some 
of the experiments we have failed in this purpose in at least one of the three time seales sinee we did not 
properly antieipate the strong effort required to respond to volunteer’s expeetations. 


Short time seale responds to a basie euriosity. This point is related to that of engagement and experiment 
set-up: The physieal presenee of seientists (with no mediation) allows to explain the experiment and 
motivate people. Also the introduetion of large sereens where the progress of other partieipants ean be 
followed on real-time helped in this matter. In the Bee-Path experiments (Bee-Path(I) and Bee-Path(2)) 
we showed the GPS loeations of the partieipants on a map, and in the Mr. Banks experiment we were 
giving a ranking list of the best players (best performanees among the partieipants). This information was 
intended to reinforee partieipation and it was also ehosen in a way that it was minimally distorting the 
questions addressed and thus not influeneing the results of eaeh experiment. Medium time seale relates 
to expeetations about the results of the experiments. Partieipants want to know whether the set-up was 
sueeessful and whether they performed well enough. In order to eomplement their experienee, it is 
important to keep them informed about the outeomes of the PUE. An example of a medium timeseale 
is the ease of the Dr. Brain where a personalized report of the performanee during the experiment was 
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sent to each participant by e-mail. In some cases and based on these results, this generated as well new 
conversations between scientists and citizens. The last time scale to manage is a more formal way of 
presenting the results of the study through public presentations and talks. This was the case for the Bee- 
Path(l) experiment, where public conferences and even a summer course dedicated to (graduated and 
undergraduated) students interested in CS practices was done. 

All the stages being discussed are important to the success of the PUEs and should be clearly exposed 
to volunteers even before they agree to participate. The return to volunteers in all these scales is a key 
ingredient to build a critical mass of engaged citizens, not only for further experiments but also to fulfil 
the non-strictly scientific objectives of the work. Being a scientist, the direct relation with volunteers 
helps to improve the message and the way of delivering the message, to refine the understanding of the 
phenomena of your experiment and even to upgrade a given experiment in future venues. A final positive 
side effect of this contact is also the raise of public awareness towards the difficulty and importance of 
science. As it can be seen, growing the project around a rich and functional webpage helps in harmonizing 
all the time scales discussed and opens new and interesting perspectives to bridge PUEs with other CS 
on-line based practices. It also serves as an efficient way to communicate results and news of the project 
to interested citizens and it could be used to improve data handling and sharing standards by allowing 
participants a direct access and management of their personal recordings. 


3 DISCUSSION 


The advent of globalization and the fast track taken by innovation (Chesbrough, 20061, combined with 
enormous challenges demand for answers at a very fast pace. Deeply interwmed global and local actions 
are necessary to face societal challenges such as the continuous growth of humanity, the effects of 
climate change or even the needs for collective decision-taking mechanisms ready for an effective policy¬ 
making. These urgencies collide with the typically long process of scienti fic research, and is affecting 


the philosophy, the resources and the methods behind the scientific method (Franzoni and Sauermann 
20141). Society expects much from us being scientists but still lacks of knowledge and reflection about 


the ways of a more collaborative, public, open and responsive research. Citizen Science practices, even 
if they may not provide definitive answers to societal challenges, expects to close the gap between public 
and researchers or in the worst case scenario to increase social awareness on the treated problems. CS 
practices can also allow science to furnish ready made solutions in public and with the active role of 
citizens. 


In this work we have presented our experience in bridging Computational Social Sciences with Citizen 
Science philosophy, which in our case has taken the form of what we call Pop-Up Experiments (PUEs). 
We hope that this work serves as guideline to groups willing to adopt and expand such practices and 
that it opens the debate about the possibilities (but also the limitations) these approaches can offer. The 
flexibility of Behavioural Experiments can be combined with the strengths of Big Data to create a new tool 
capable to generate a new collective knowledge. We have conceptually identified the main issues to take 
into account whenever planning a CS research into CSS field. We have grouped the challenges into three 
categories according to our experience: Engagement, Infrastructure and Return. Eurthermore, we have 
explained the solutions we implemented under the framework of a PUE, providing practical examples 
grounded in our own experience. The importance of team work and of widening the scope to consider 
questions not directly related with lab work has been highlighted as well as the need to work hand in hand 
with both public and other social actors. Other technical aspects of the approach have also been reviewed. 


We have supported the idea of abandoning the Ivory Tower and opening up science and its research 
processes. Indeed, CS research essentially relies in the citizenship collaboration, but not just in the 
passive data gathering. We think that to place Distributed Intelligence (with contributions from experts 
and amateurs) in the very core of scientific analysis can be also a valid strategy for obtaining rigorous 
and valuable results. We also believe that the PUEs we herein present can potentially emp ower citizens 


to take their own civic actions relying on a collectively constructed facts-based approach (McQuillan 


Frontiers 


9 










Sagarra et al. 


Citizen Science practices 


20141. To co-create and co-design with citizens a smart city along the Big Data paradigm will then be 
much easier and even more natural since we will share interests and concerns during the whole research 
process. Data gathered in the wild or in-vivo contexts could be thus understo od as truly public and open 
while data ownership and knowledge would be shared from the very start ({Callon and Rabeharisoa 
2003]). Our future venues and experienc es will t ake de eper inspiration from Open Source, do-it-yourself. 


do-it-together and makers movements ([Hatch] |2014|) which facilitates learning-by-doing, low-cost and 
heuristic skills to everybody. A fresh look to problems can contribute with innovative and imaginative 
ideas that at the end can reach out of the box solutions. However, being scientists, we will also have to find 
the way to reconcile unorthodox and intuitive forms with the standards and methodologies of the world 
of science. Lessons shall be learnt from the Open Prototyping approach where an industrial product such 
as a car can be shaped by an iterative proce ss where the company o wning the product has no problem 


22 _ 

to allow for inputs coming from the outside ( jBullinger et al.[ [201 Ij ). Some other clues can be found in 


the form of collective experimentation where a fruitful dialogue can be stablished among the matters of 


concern raised by citizens and the matters of fact raised by scientists. Latour (2009) already introduced 
these concepts and discussed the symbiotic relationship by taking the case study of Ecologism (a civic 
movement) and Ecology (a scientific activity). There are still many aspects to test and explore following 
this approach within the CSS research field. 

We would also like however to shortly present open questions related to the way we perform science 
now adays, and echo fundamental contr adictions that science is not properly handling in this globalization 
era ([Franzoni and Sauermanii||2014|). Citizen practices yield nice outputs for the communities, yet they 
require a lot of effort by scientists, with the downside of providing very low (formal and bureaucratic) 
professional recognition. Open social experiments demand a high level of involvement in cooperation 
with non-scientifflc actors which may divert professional researchers from the activity for which they 
are evaluated: publication of results. Eurthermore, such experiments often involve multidisciplinar teams, 
which then may find difficulties finding the appropriate journals to publish, facing difficult acceptance in 
stablished communities. We thus urge scientific community to actively recognize the valuable advantages 
of performing science with our proposed experimental framework. 


Lessons learned must be shared, both within the community (like this paper is trying to do) and outside 
in public spaces including also public institutions and policy makers. Science and Citizen Science are 
mostly publicly funded and therefore belong to society. Internet provided new ways in which new relations 
among science and society can be strongly reinforced. We believe that this is good for everyone by raising 
Science concern, by enhancing participation and, most importantly, by exploring new effective ways to 
push knowledge further. We hope that the present work helps in theoretically establishing the concept of 
Pop-Up Experiments and encouraging the adoption of CS practices in science, in whichever the field. 


4 MATERIAL & METHODS 

In this section we provide description of our experience throughout four years of experiments in 
performing pop-up events using CS practices in the city of Barcelona. Here we detail the experimental 
methods and briefly summarize the outcome. All experiments have been performed in accordance with 
institutional (from the University of Barcelona in all cases except for Cooperation! 1) experiment which 
was from Universidad Carlos III) and national guidelines and regulations in data privacy (following the 
Spanish law EOPD). All interfaces being used included an informed consent from all subjects. Data 
collected was properly anonymized and unrelated from personal details which in our case was age range, 
genre, level of studies and electronic address (e-mail). 

4.1 BEE-PATH(I) EXPERIMENT 

The aim of this experiment is to study the movement during the exploration of an outdoor science and 
technology fair where several stands with activities were located in an area of approximately 3 ha inside 
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a public green park. The experiment took plaee during the weekend of 16th and 17th of June of 2012, 
speeifioally in Saturday afternoon (from 16h to 20h) and the morning of Sunday (from llh to 15h). 
The partieipants had very different interests, origins, baekground and ages and organisation of the event 
estimated that 10,000 people visited the fair. The Bee-Path information stand was loeated at the main 
entranee where visitors were eneouraged to partieipate in the experiment by downloading an App from 
their mobile phones. After a very simple registration and instruetions to aetivate the App, partieipants 
were left wandering around the fair while being traeked. After a laborious eleaning proeess we eould 
analyse the movement and trajeetories of 27 subjeets among those reeords provided by 101 volunteers. We 
found spatio-temporal patterns in the movement and we developed a theoretieal model based in Langevin- 
dynamies in a gravitational potential landseape ereated by the stands. This model based was able to 
explain the results of the experiment and to model different s eenarios with other spatml eonfi gurations of 
the stands. Seientifie paper is been submitted for publieation (|Gutierrez-Roig et al4|2015b|). The projeet 
deseription, results and data are freely aeeessible on the webpage www.bee-path.net, 


4.2 BEE-PATH(2) EXPERIMENT 

Following the previous Bee-Path(l) experiment this setup was also foeussed in studying the movement, 
but in this ease we were interested in human searehing patterns or how people move when explores a 
landseape to find something. The experiment took plaee in the next edition of the same seienee and 
teehnology fair (June 15th and 16th of 2013). Partieipants also downloaded a similar App that were 
traeking them, but in this ease were instrueted to find 10 mannequins hidden in the park. Numerous 
problems were eneountered due to several teehnologieal and non-teehnologieal faetors whieh impeded 
a satisfaetory performanee of the experiment. Teehnologieal troubles were prineipally two; The low 
aeeuraey of reeordings due to the proximity of regional parliament where the wi-fi and mobile phone 
eoverage is inhibited, and the limitation of the App to perform adequately when running in low-end 
deviees. Non-teehnologieal part ineludes a bad plaeement (far from the entranee) of our stand where the 
reeruitment of volunteers was hard and the unexpeeted faet that some vandals ehanged, stole and even 
manipulated the mannequins’ position. Notwithstanding the failure, we learnt important lessons from 
these eomplieations. 


4.3 COOPERATION(I) EXPERIMENT 


Here we explored how important is the age in the emergenee of eooperation when people are faeing 
repeated Prisoner Dilemma. The experiment was earried out with 168 volunteers seleeted from the 
attendants to DAU Bareelona Festival 2012 (1st Board Game Fair of Bareelona, Deeember 15th and 
16th). This volunteers’ set was divided into 42 subsets of 4 players aeeording to the age, seven different 
age groups plus one eontrol group where subjeets where not distinguished by age. Eaeh subset made 
up a game where the four partieipants played 25 rounds (although they were not aware of it) deeiding 
between two eolors assoeiated to a eertain prisoner’s dilemma payoff matrix. Partieipants played a 2x2 
PD game with eaeh of their 3 neighbors, ehoosing the same aetion for all opponents. In order to play 
with an ineentive, they were remunerated with real money proportional to the final seore. During the 
game volunteers interaeted through software speeially programmed for the experiment and installed in 
a laptop. They were not allowed to talk or signal in any way, but to further guarantee that potential 
interaetions among players would not influenee the results of the experiment, the assignment of players 
to the different eomputers of the room was eompletely random. In this experiment, together with the 
Jesuites Gasp deseribed in the next subseetion, we found that the elderly eooperate more and there exist a 
behavioural transition from reeiproeal, but mo re volatile behaviour to more persistent aetions towards the 
end of adoleseenee. For further details eonsult Gutierrez-Roig et al. |2014). 
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4.4 C00PERATI0N(2) EXPERIMENT 


The purpose to repeat the Cooperation(l) experiment at DAU festival was to eonfirm the apparent tendeney 
of ehildren to eooperate less than the averaged population. Thus, we repeated the experiment just to 
augment the pool of subjeets from this age range what allowed us to be more statistieally aeeurate. We 
analysed the performanee of 52 students of seeondary sehool (Jesuites Casp experiment) ranging from 12 
to 13 years old. The methods and protoeols were the same as in Cooperation(l) experiment, as well as the 
software installed in its own laptops. The results of this experiment rejeeted the hypothesis that ehildren 
eooperated less in average b ut eonfirmed at the same time their more volatile behaviour, as it deseribes 


Gutierrez-Roig et al. (2014). 


4.5 MR.BANKS(1) EXPERIMENT 


Mr. Banks experiment was set up to study how non-expert people make deeisions in uneertain 
environments, eoneretely assessing their performanee when trying to guess if a real finaneial market 
priee would go up or down. We analysed the performanee of 283 volunteers (from the approximately 
6,000 attendants) to DAU Bareelona Festival 2013 on Deeember 14th and 15th. All the volunteers played 
via an interfaee speeifieally ereated for the experiment that was aeeessible through identieal tablets only 
available in a speeifie room under researeher’s surveillanee. Deviees showed in the main sereen the historie 
of daily market priee eurve and some other information like 5-day and 30-day average windows eurves, 
high-frequeney priee of previous day, the opinion of an expert, priee direetion of previous days and priee 
direetions of other markets around the world. All priee eurves and information was extraeted from real 
historieal series. Partieipants eould play in four different seenarios with different time and information 
availability eonstrains. In eaeh seenario they were required to make guesses during 25 rounds while every 
eliek in the sereen was reeorded. Eaeh player started with 1,000 eoins and he/she earned 5% over their 
eurrent seore if the guess was eorreet or got a negative return of the same size if he/she was wrong. We 
used gammifieation strategies and we did not provide any eeonomie ineentive in eontrast with Soeial 
Dilemma experiments. The analysis of 18,436 reeorded deeisions and 44,703 elieks allowed us to elaim 
that partieipants tend to follow intuitive strategies ealled Market Imitation and Win-Stays Lose-Switeh. 
These str ategies are less followed when there is more time to make a deeision or some information is 


provided (|Gutierrez-Roig etal 

at WWW. mr-banks . net 


2015a I. Either the experiment and information of the projeet is available 


4.6 MR.BANKS(2) EXPERIMENT 

Here we repeated the Mr.Banks(l) experiment in a different eontext in order to study the reprodueibility 
of the results obtained. The interfaee and the experimental set-up were the same as in Mr.Banks(2) but the 
typology of partieipants and the type of event were signifieantly different. The experiment, named Haek 
your Brain in the eonferenee programme, was situated at the main entranee of the CAPS2015 eonferenee, 
the International Event on Colleetive Awareness Platforms for Sustainability and Soeial Innovation, held 
in Brussels (July 7th and 8th of 2015). The 42 volunteers that played the game provided 2,372 reeorded 
deeisions. The volunteers were all registered partieipants of the eonferenee with very diverse profiles 
(seientists mostly from soeial seienees, soeial innovators, designers, soeial entrepreneurs, poliey-makers, 
ete.). The results showed that the results of the Mr.Banks Experiment had a good reprodueibility as the 
pereentage of eorreet guesses were similar in Mr.Banks(l) and Mr.Banks(2) (53,43 % and 52,74 %, resp.) 
as well as the pereentage of market-up deeisions (60,83 % and 60,48 %, respeetively). A deeper analysis 
of the results is underway to eheek if the strategies used by Mr.Banks(l) volunteers are the same than 
those used by Mr.Banks(2) volunteers. 
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4.7 DR.BRAIN EXPERIMENT 


In this work, we report the results of a lab-in-the-field experiment that allows for a phenotypie 
eharaeterization of individuals when faeing different soeial dilemmas. Instead of playing with the same 
fixed payoff matrix as in Cooperation experiment, here the values and the neighbors ehanged every round. 
We diseretized the (T, S')-plane as a lattiee of 11 x 11 sites allowing us to explore up to 121 different 
games grouped in 4 eategories: Harmony Game, Stag Hunt Game, Snowdrift Game and Prisoner Dilemma 
Game. Each player was given a tablet with the application of the experiment. The participants were shown 
a brief tutorial, but were not instructed in any particular way nor with any particular goal in mind. They 
were informed that they had to make decisions in different conditions and against different opponents in 
every round. Due to practical limitations, we could only host around 25 players simultaneously, so the 
experiment was conducted in several sessions over a period of two days. In every session, all individuals 
played a different, randomly picked number of rounds between 13 and 18. The total number of participants 
in our experiment was 541, adding up to a total of 8, 366 game actions collected. In order to play with 
an incentive, they received back coupons for a prize of 50 euros to spend in stores of the neighborhood. 
During the game volunteers interacted through software specially programmed for the experiment and 
installed in a laptop. They were not allowed to talk or signal in any way, and again were placed spatially 
at random. This experiment concludes that we can distinguish, empirically and without making any 
assumption, five different player’s behaviour or phenotype s not theoretically predicted. Paper is been 
submitted for publication ( [Poncela-Casasnovas et al.[|2015| ). 
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Table 1. Summary of our seven experiments to test Citizen Seienee praetiees with Pop-Up 
Experiments in urban eontexts and along the Computational Soeial Seiences researeh. First 
eolumn provides the different aspeets that each of the experiments has considered in terms of 
Experimental Infrastructure and Set-Up, Public Engagement tools and Outcome and Return to the 
Public. The other columns specify how the different experiments consider these aspects. Further 
details are provided in Material and Methods section while Sect, [^deeper discuss the different aspects. 


Experiments 

Bee-Path(l) 

Bee-Path(2) 

Cooperation(l) 

Cooperation(2) 

Mr. Banksll) 

Mr. Banks(2) 

Dr. Brain 

l.General Information 

Date 

June 2012 

June 2013 

December 2012 

March 2014 

December 2013 

July 2015 

December 2014 

Topic 

Human Mobility 

Human Mobility 

Social Dilemmas 

Social Dilemmas 

Decision Making 

Decision Making 

Social Dilemmas 

Number of volunteers 

101(27 valid) 

68 

168 

52 

283 

42 

541 

Number of records 

10,312 GPS points 

10,315 GPS points 

4,200 decisions 

1,300 decisions 

18,525 decisions 

2,372 decisions 

8,366 decisions 

Budget (approx.) 

4,000 EUR 

3,500 EUR 

2,000 EUR 

1,000 EUR 

2,500 EUR 

1,300 EUR 

1,000 EUR 

2.Experlmental Infrastructure and Set-Up 

Hosting Event 

Science Fair 

Science Fair 

Games Festival 

Secondary School 

Games Festival 

Conference 

Games Festival 

Space 

Outdoor Booth 

Outdoor Booth 

Room 

4 Classrooms 

Habilitated space 

Habilitated space 

Habilitated space 

(m‘) 

(9m") 

(18m") 

(20m") 

(200m") 

(80m") 

(20m") 

(60m") 

Duration 

1,5 day (weekend) 

1,5 day (weekend) 

1,5 day (weekend) 

0,5 day 

2 days (weekend) 

2 days 

2 days (weekend) 

Typoiogy of volunteers 

Science fair visitors 

Science fair visitors 

Festival visitors 

School pupils 12-13 Festival visitors 

Conference 

Festival visitors 

(free entrance) 

(free entrance) 

(free entrance) 

years 

(free entrance) 

participants 

(free entrance) 

Recording Device 

Mobile phone 

Mobile phone 

Laptops 

Laptops 

Tablets 

Tablets 

Tablets 

S.Publlc engagement tools 

Number of scientists and 
facilitators 

8 

7 

7 

4 

6 

4 

10 


Official event 

Official event 

Official event 


Official event 

Official event 

Official event 


program 

program 

program 


program 

program 

program 

Preparatory actions 

Webpage 

Webpage 



Character and 

platform creation 


Character and 

platform creation 


Press Release 




Press release 


Press release 


Logo Design 




Targeted mailing 


Targeted mailing 

Twitter campaign 

On-site visibility 

Location at the 

entrance 

Location far from 

the entrance 

Isolated room 

- 

Location in the 

main hall 

Location at the 

main entrance 

Location on main 

entrance path 


Panel 

Panel 

Panels 


Panels 

Panel 

Panels 


Flyers 

Flyers 

Actor 


Flyers 

Flyers 

Flyers 

On-site promotion tools 

Screen (real-time Screen (real-time 
paths visualization) paths visualization) 



Screen with players 
ranking 

Merchandising 

Screen with 

announcement 



Actors 



Actor 


Actor 






Merchandising 


Merchandising 

Scenography 

4.0utcome and Return to the public 

Scientific paper 

Submitted 

None 

Published 

Published 

Submitted 

Submitted 

Submitted 

Sci. Communications 

2 

None 

4 

4, same as Coop(l) 2 

Not yet 

Not yet 

Website 

bee-path.net 

bee-path.net 

nectunt.bifi.es 

nectunt.bifi.es 

mr-banks.net 

mr-banks.net 

nectunt.bifi.es 

OpenData 

Data and software 

Data and software 

- 

- 

Data available 

Data available 

In preparation 


On-site discussion 

On-site discussion 

On-site discussion 

Coupons for books 

On-site discussion 

On-site discussion 

On-site discussion 


with scientists and 

with scientists and 

with scientists 

and office material 

with scientists 

with scientists 

with scientists 

Short-time return 

data visualization 

data visualization 



Ranking 

Ranking 





Economic reward 


visualization 

visualization 

Coupons for a prize 


Personalized 

Personalized 


Letter to parents 



Personalized 

Medium-term return 

report sent by 

report sent by 

- 

Letter to school 

- 

- 

report sent by 


email 

email 





email 


Public event to 

present the results 

and Courses 

Public event to 

present the results 

and Courses 

Targeted mailing 

Targeted mailing 




Long-term return 

about scientific 

publication and 

results 

about scientific 

publication and 

results 

In preparation 

In preparation 

In preparation 
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Figure 1. Research process in Pop-up Experiments. We identify the different steps of the whole process 
in a set of boxes, from its conception to its completion. Starting from the research question, we introduce 
the design and the performance of the given experiment aiming to respond to a particular challenge. 
Crowsourced data gathered can be analyzed both by expert scientists and citizen amateurs to produce new 
knowledge (return) in multiple forms. The results are finally taken as inspiration and renewed energy to 
face new challenges and new research questions. Data quality is also part of the process: lessons learnt 
from the scientific analysis can improve future experimental conditions. Oval forms in orange correspond 
to volunteers’ contributions, squared forms in magenta are tasks exclusively done by scientists while 
rounded rectangles in green are shared tasks by both citizens and scientists. 
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microscale 


mesoscale 


macroscale 



Short-time Short-time Short-time 



Behavioural Experiments Pop-Up Experiments 


Big Data 


Figure 2. Radar Chart comparing seven eharaeteristics of the Pop-Up Experiment approaeh with the 
research made through Behavioural Experiments and Big Data. Eeatures are graded in three different 
degrees of intensity from low (shortest radius) to high (largest radius). ’’Short-time” describes the time 
required for running the experiment. ’’Scalable” qualifies how easy is to upseale and augment the number 
of subjeets in the experiment while preserving original design. ’’Universal” quantifies the generality of 
the statements produeed by the experiments. ”In-vivo” measures how elose is the experimental set-up 
to quotidian situations or everyday life situations. ’’Reprodueible” assesses the capaeity to repeat same 
experiment under identical conditions. Einally, ’’Tunable” quantifies how flexible and versatile are the 
conditions of the experimental design. 
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